Real time assessment of picture quality

ABSTRACT

A computerized method for computing the photo quality of a captured image in a device image acquisition system, comprising on-board combining of a plurality of quality indicators computed from said captured image and its previous image frames quality indicators and a confidence level for at least one of said quality indicators; and using a processor to determine, based on said combining, whether photo quality is acceptable and taking differential action depending on whether quality is or is not acceptable.

RELATED APPLICATIONS

This application is a continuation application of U.S. application Ser.No. 15/582,722 filed on Apr. 30, 2017, which is a continuationapplication of U.S. application Ser. No. 15/007,253 filed on Jan. 27,2016 (U.S. Pat. No. 9,661,226, issued May 23, 2017), which is acontinuation application of U.S. application Ser. No. 14/437,105 filedon Apr. 20, 2015 (U.S. Pat. No. 9,338,348, issued May 10, 2016), whichis a national phase of, and claims priority from, PCT Application No.PCT/IL2013/050856 filed on Oct. 22, 2013, which claims priority fromU.S. Provisional Application No. 61/717,216 filed on Oct. 23, 2013 andU.S. Provisional Application No. 61/759,643 filed on Feb. 1, 2013, allof which applications are hereby incorporated by reference as fully setforth herein.

FIELD

The present invention relates generally to image processing and moreparticularly to analysis of photographed images.

BACKGROUND

Cameras today, even those with good “auto mode” functions, still rely onthe camera user to assess a picture's quality, either at the time oftaking the picture, or at a later stage. Certain software, such asPhotoshop, offers tools to enhance a picture to an extent. Many userssimply take more than one picture, say 5-6 pictures, so they will beable to choose one good one.

Conventional technology constituting background to certain embodimentsof the present invention is described in the following publicationsinter alia:

a. US 20130155474 in which a user of the mobile device is provided withfeedback prior to the capturing of at least one image of a paperdocument which provides the user with at least one instruction foradjusting at least one measured parameter. Parameter values may becombined into a group threshold or total overall quality score, suchthat the combination of all of the measured parameter values produces anoverall quality score which exceeds a defined threshold value before theimage can be captured by the camera. According to this publication, “Theuser can be provided detailed information to assist the user in taking abetter quality image of the document. For example, the blurriness mayhave been the result of motion blur caused by the user moving the camerawhile taking the image. The test result messages can suggest that theuser hold the camera steadier when retaking the image.”

b. WO 2006040761/US20070195174 which describe a system in whichinterface module further enables to define the scene dynamics of thecaptured image and attributes of the captured image scene dynamicsinclude image motion speed and motion speed of the subjects and theinterface module further enables setting the captured image attributesrelative weight for the compution of the total image grade.

c. US20090278958 which describes inter alia that “The scoring of acurrent base image may be based on scores which have been given topreviously captured base images. In such a manner, redundant calculationmay be avoided.”

d. Other systems such as the following patent documents: U.S. Pat. No.7,362,354; WO2003069559; U.S. Pat. No. 7,688,379; US20090263028;US20060017820; US20060056835; U.S. Pat. No. 8,125,557; WO2011148212;U.S. Pat. No. 7,920,727; US20130155474; U.S. Pat. No. 8,040,382.

The disclosures of all publications and patent documents mentioned inthe specification, and of the publications and patent documents citedtherein directly or indirectly, are hereby incorporated by reference.Materiality of such publications and patent documents to patentabilityis not conceded

SUMMARY

Certain embodiments seek to utilize camera hardware such asaccelerometers, gyros and autofocus lenses, inter alia, to evaluatepictures taken in real time, and actively assist in obtaining the bestpicture given the circumstances at hand.

Typically, each available sensor, e.g. accelerometers, gyros, sends itsraw data to a processing component 400 that outputs a quality indicatore.g. as illustrated in FIG. 1, and FIG. 3. Data may also be sent toDevice/surrounding objects 4D position computation component 230 for 4Dpositioning computation. Raw data may be combined with qualityindication from other sensors, such as the camera/lens module 200, thatcan send its own quality indications. For example, some lens modules canestimate the picture focus quality, as it is a part of their auto focusfunctionality. Some camera/lens modules may have dual cameras or morefor achieving 3D data, sophisticated focusing methods like Lytro lightfield camera, or their own hardware sensors like Google Glass. All thisavailable data can be sent to component 400 and Device/surroundingobjects 4D position computation component 230 for use of theirfunctions. Various quality indicators, depending on device capabilitiesand implementation, may be employed.

Prior art has used certain independent quality indicators, each giving aquality of one particular aspect of a picture, for a given picture. Sometry to quantify the quality by means of a total quality indicator.Typically this is done as follows:

${QI}_{total} = {\sum\limits_{i = 1}^{n}{{QI}(t)}_{i}}$Where the number of quality indicators (QI), and t is the time of test.

Some implementations use a weight function in computation, but theweight is either constant or can be change by the user manually.

${QI}_{total} = {\sum\limits_{i = 1}^{n}{{{QI}(t)}_{i}*{w_{i}(t)}}}$

However, in this invention, the weight of one indicator will take intoaccount data from other quality indicator/s e.g. their quality indicatorvalue, weight, confidence level (explained forthwith) in them and theirprevious value, weight and confidence level. For example, if both deviceshake quality indicator and/or camera focus quality indicator indicate apoor quality, disregard use of aesthetic quality indicator in totalcomputation, even if user indicated that he would like to give it a highpriority. This will prevent the device from taking a picture when thepicture is blurry, as is the case in prior art. Once shake is low andfocus is optimal, aesthetic QI may be used in total qualitycomputations. One quality indicator's importance or relative weightwithin the total quality indicator may change over time, if some otherquality indicator indicates that the first quality indicator becomesless relevant in quantifying the picture quality.

Furthermore, since each QI algorithm has its own assumptions, and eachsensor may have measurement errors, the quality indication (QI) valuemay therefore not be reliable. Also, a QI value that fluctuates widelymay not be reliable, and in certain cases, re-assessment of the qualityindicator may be called for. Thus, a “confidence level” in the QI isobtained. These indicators may be relayed to a processing component thatanalyses the current relevance of each indicator, and with suitableconfiguration, provides a “total” quality value.

This total quality indicator and its detailed list of quality indicatorscan be used to give the user accurate feedback on the picture quality,whether and how it fits user preferences (configured in 151, used intotal QI computation), and provide detailed suggestions on how to take abetter picture, if possible.

Suggestions may be provided as an oral message, typically pre-recorded,as an icon on a screen, or as text. Suggestions may use the recordedimages as a template for the suggestions. This may include marking overexposed areas in red, or marking a problematic element with a circle.

The quality indicators' values and total quality indicator can be shownto the user in real time.

Certain embodiments of the present invention include a method whichcomprises combining data from multiple sensors and multiple qualityindicators, using available GPU and CPU, to compute, in real time, eachquality indicator, and to form a combined total indicator to be shown tothe user before and after he has pressed the shutter button. The usermay get, or choose to get, suggestions from the application on how toimprove the next shot, using the quality indicators and theircorrelation.

Further indication to the user can be made if the image is below acertain total quality indicator threshold.

According to one embodiment, quality indications are used to control thelens module 200. For example, focus distance reported may be below hyperfocal distance or very short and accelerometer reports movement indirection perpendicular to focus plan. This can be used to find a way toshorten the shutter speed (increase ISO or change aperture for example).Another example may be to change the focus point and possibly focusdistance if the object being tracked has moved.

In another embodiment, when the user presses the “shutter” button totake the picture (component 105), the device will wait for the totalquality indicator to reach a minimum value/threshold, before taking afull resolution picture as shown in FIG. 1. Ideally, this time will beclose to the shutter release time. A timeout mechanism can ensure a fullresolution picture will be taken, even if the threshold is not reachedin a given time after the shutter is released.

A list of suggestions to the users may be for instance “blurred imagedue to camera shake”, or “blurred subject due to subject movement”, or“Image dynamic range is beyond the sensor dynamic abilities—choose areaof interest, or take a high dynamic range (HDR) shot”.

Embodiments include but are not limited to:

Embodiment 1. A method for computing the photo quality of a capturedimage in a device image acquisition system, said method comprising:

-   -   computing at least one total quality indicator based on at least        two quality indicators computed from said captured image and its        previous image frames,    -   to determine whether the photo quality is acceptable; and    -   providing a corresponding detailed photo quality feedback.

Embodiment 2. A system according to embodiment 1, where automaticallyactivating the capturing apparatus, once a logical criterion based onsaid quality indicators is met.

Embodiment 3: A system, according to embodiment 1 where

-   -   a message provider is operative where at least one appropriate        suggestion from a pre-stored table of suggestions on how the        digital image can be improved further,    -   where said message is presented as a suggestion to the user.

Embodiment 4: A system, according to embodiment 1 where

-   -   pictures are taken and saved at each frame, but only pictures        with a said logical criterion are marked as so.

Embodiment 5: A system, according to embodiment 1 where the stream offrames is saved as a video and;

-   -   quality indicators, user preferences, confidence level, and        quality importance are used to quantify video quality.

Embodiment 6: A system, according to embodiment 5, used to give a usersuggestions on improvement of the video.

Embodiment 7: A system, according to embodiment 1, where computed QI andtheir confidence levels are used to change parameters in the lens/sensormodule to achieve better total QI.

Embodiment 8: A system and method as in embodiment 1 used for automaticimage capturing of at least one picture once a configured total qualityindicator meets a minimum threshold, even without the user pressing ashutter button.

Embodiment 9. A system using blur type detection for alerting the userof object movement in a region.

Embodiment 10. A system using blur type detection for alerting the userhe is moving too fast in a vehicle to achieve a non-blurred image.

Embodiment 11. A system and method described herein for evaluating videoquality.

Embodiment 12. A system with device shake recognition, where a devicemovement gesture can be used for shutter press indication.

Embodiment 13. A system with device shake recognition, where a devicemovement gesture can be used for delete picture indication.

Also provided, excluding signals, is a computer program comprisingcomputer program code means for performing any of the methods shown anddescribed herein when said program is run on a computer; and a computerprogram product, comprising a typically non-transitory computer-usableor -readable medium e.g. non-transitory computer-usable or -readablestorage medium, typically tangible, having a computer readable programcode embodied therein, said computer readable program code adapted to beexecuted to implement any or all of the methods shown and describedherein. It is appreciated that any or all of the computational stepsshown and described herein may be computer-implemented. The operationsin accordance with the teachings herein may be performed by a computerspecially constructed for the desired purposes or by a general purposecomputer specially configured for the desired purpose by a computerprogram stored in a typically non-transitory computer readable storagemedium. The term “non-transitory” is used herein to exclude transitory,propagating signals or waves, but to otherwise include any volatile ornon-volatile computer memory technology suitable to the application.

Any suitable processor, display and input means may be used to process,display e.g. on a computer screen or other computer output device,store, and accept information such as information used by or generatedby any of the methods and apparatus shown and described herein; theabove processor, display and input means including computer programs, inaccordance with some or all of the embodiments of the present invention.Any or all functionalities of the invention shown and described herein,such as but not limited to steps of flowcharts, may be performed by aconventional personal computer processor, workstation or otherprogrammable device or computer or electronic computing device orprocessor, either general-purpose or specifically constructed, used forprocessing; a computer display screen and/or printer and/or speaker fordisplaying; machine-readable memory such as optical disks, CDROMs, DVDs,BluRays, magnetic-optical discs or other discs; RAMs, ROMs, EPROMs,EEPROMs, magnetic or optical or other cards, for storing, and keyboardor mouse for accepting. The term “process” as used above is intended toinclude any type of computation or manipulation or transformation ofdata represented as physical, e.g. electronic, phenomena which may occuror reside e.g. within registers and/or memories of a computer orprocessor. The term processor includes a single processing unit or aplurality of distributed or remote such units.

The above devices may communicate via any conventional wired or wirelessdigital communication means, e.g. via a wired or cellular telephonenetwork or a computer network such as the Internet.

The apparatus of the present invention may include, according to certainembodiments of the invention, machine readable memory containing orotherwise storing a program of instructions which, when executed by themachine, implements some or all of the apparatus, methods, features andfunctionalities of the invention shown and described herein.Alternatively or in addition, the apparatus of the present invention mayinclude, according to certain embodiments of the invention, a program asabove which may be written in any conventional programming language, andoptionally a machine for executing the program such as but not limitedto a general purpose computer which may optionally be configured oractivated in accordance with the teachings of the present invention. Anyof the teachings incorporated herein may wherever suitable operate onsignals representative of physical objects or substances.

The embodiments referred to above, and other embodiments, are describedin detail in the next section.

Any trademark occurring in the text or drawings is the property of itsowner and occurs herein merely to explain or illustrate one example ofhow an embodiment of the invention may be implemented.

Unless specifically stated otherwise, as apparent from the followingdiscussions, it is appreciated that throughout the specificationdiscussions, utilizing terms such as, “processing”, “computing”,“estimating”, “selecting”, “ranking”, “grading”, “calculating”,“determining”, “generating”, “reassessing”, “classifying”, “generating”,“producing”, “stereo-matching”, “registering”, “detecting”,“associating”, “superimposing”, “obtaining” or the like, refer to theaction and/or processes of a computer or computing system, or processoror similar electronic computing device, that manipulate and/or transformdata represented as physical, such as electronic, quantities within thecomputing system's registers and/or memories, into other data similarlyrepresented as physical quantities within the computing system'smemories, registers or other such information storage, transmission ordisplay devices. The term “computer” should be broadly construed tocover any kind of electronic device with data processing capabilities,including, by way of non-limiting example, personal computers, servers,computing system, communication devices, processors (e.g. digital signalprocessor (DSP), microcontrollers, field programmable gate array (FPGA),application specific integrated circuit (ASIC), etc.) and otherelectronic computing devices.

The present invention may be described, merely for clarity, in terms ofterminology specific to particular programming languages, operatingsystems, browsers, system versions, individual products, and the like.It will be appreciated that this terminology is intended to conveygeneral principles of operation clearly and briefly, by way of example,and is not intended to limit the scope of the invention to anyparticular programming language, operating system, browser, systemversion, or individual product.

Elements separately listed herein need not be distinct components andalternatively may be the same structure.

Any suitable input device, such as but not limited to a sensor, may beused to generate or otherwise provide information received by theapparatus and methods shown and described herein. Any suitable outputdevice or display may be used to display or output information generatedby the apparatus and methods shown and described herein. Any suitableprocessor may be employed to compute or generate information asdescribed herein e.g. by providing one or more modules in the processorto perform functionalities described herein. Any suitable computerizeddata storage e.g. computer memory may be used to store informationreceived by or generated by the systems shown and described herein.Functionalities shown and described herein may be divided between aserver computer and a plurality of client computers. These or any othercomputerized components shown and described herein may communicatebetween themselves via a suitable computer network.

BRIEF DESCRIPTION OF THE DRAWINGS

Certain embodiments of the present invention are illustrated in thefollowing drawings:

FIG. 1 is a functional block diagram of a system and method for realtime assessment of picture quality produced on a device, e.g. as shownin FIG. 2, having multiple sensors including use of hardware sensor datato generate and utilize picture quality indications.

FIG. 2 illustrates a hardware and operating system (OS) layer which isoperative in conjunction with the functional block diagram of FIG. 1.

FIGS. 3 and 4 illustrates component 400 in detail and a sample of aquality indicator computation dependence, all according to certainembodiments.

Computational components described and illustrated herein can beimplemented in various forms, for example, as hardware circuits such asbut not limited to custom VLSI circuits or gate arrays or programmablehardware devices such as but not limited to FPGAs, or as softwareprogram code stored on at least one tangible or intangible computerreadable medium and executable by at least one processor, or anysuitable combination thereof. A specific functional component may beformed by one particular sequence of software code, or by a plurality ofsuch, which collectively act or behave or act as described herein withreference to the functional component in question. For example, thecomponent may be distributed over several code sequences such as but notlimited to objects, procedures, functions, routines and programs and mayoriginate from several computer files which typically operatesynergistically.

Data can be stored on one or more tangible or intangible computerreadable media stored at one or more different locations, differentnetwork nodes or different storage devices at a single node or location.

It is appreciated that any computer data storage technology, includingany type of storage or memory and any type of computer components andrecording media that retain digital data used for computing for aninterval of time, and any type of information retention technology, maybe used to store the various data provided and employed herein. Suitablecomputer data storage or information retention apparatus may includeapparatus which is primary, secondary, tertiary or off-line; which is ofany type or level or amount or category of volatility, differentiation,mutability, accessibility, addressability, capacity, performance andenergy use; and which is based on any suitable technologies such assemiconductor, magnetic, optical, paper and others.

DETAILED DESCRIPTION OF THE DRAWINGS

Certain embodiments of the present invention include a system and methodfor real time assessment of picture quality produced on a device havingmultiple sensors, such as a smartphone. In one embodiment, all of thesensors, such as but not limited to an image sensor, accelerometer orgyroscope, are embedded in one device. It may be that they reside indifferent devices, but send their data to one device where the totalindicators are computed.

FIG. 1 is a functional block diagram of a system and method for realtime assessment of picture quality produced on a device, e.g. as shownin FIG. 2, having multiple sensors including use of hardware sensor datato generate and utilize picture quality indications.

Referring now to FIG. 1, a Lens/sensor module 200 such as OmniVisionOM8865 may be characterized by and/or generate photographic parameters,including but not limited to some or all of:

-   -   a) The current image the sensor receives, referred to as a frame        or image    -   b) ISO speed (current, min, max)    -   c) Is camera focused (yes/no/estimate if available).    -   d) Aperture    -   e) Focal length    -   f) Shutter speed    -   g) Focus rectangle, or area of interest    -   h) Sensor type and size—for example BSI2 CMOS 1/2.5 inch    -   i) Depth of Field    -   j) Scene mode or user preferences.        -   These parameters may be used for computing quality indicator            values and their confidence level, as indicated in FIG. 3.

The module 200 sends continuous video frames of the scene in front ofthe device lens, and data from other sensors is continuously receivedand analyzed in real time.

Typically, a smartphone includes all such sensor and hardware inside thesmartphone. However, it is possible that the image may be sent to thedevice from an external camera via a cable or Wi-Fi. The image sensorsends its data to the device operating system that may send the wholeimage or a preview of it to the GPU 122 of FIG. 2 at a rate of X framesper second. This image data is passed on to various programmed GPUvertex and fragment shaders. It is then used for fast computation ofsome of the quality indicators.

FIG. 2 illustrates a hardware and operating system (OS) layer, which isoperative in conjunction with the functional block diagram of FIG. 1.

Errors and Probability in Quality Indicators

When prior art computes a quality indicator value, it was done withouttaking into account the possibility of error in the computed QI, as allsensors give out errors.

For example, a user wishes to compute a quality indicator for deviceshake based on data from accelerometers. Accelerometers have precisionerrors and drifting errors over time. If the user wishes to compute aspeed vector, this will be relevant. The speed may be assumed to be100+−3. A GPS can estimate location and speed errors via its locationcomputation. Errors may also occur as a result of blurriness of somesort, or may result from image resolution and many other factors. Forexample, assuming a device is moving at a constant speed, and V is thecomputed speed in frame i, where 1≤i≤n, then if value V_(n) is 2standard deviation from the average speed measured so far, (and assuminga normal distribution), then probability for this value is 5%, denotedP_(i)(t). Most of the time the speed may not be constant, but the abovetechnique can nonetheless can be applied, mutatis mutandis, when errorscan be estimated. The probability factor P, is closely related to thenature of the quality indicator and hardware used. For example, a GPSdevice has its own algorithm for that e.g. Kalman Filters. Beyond mereerrors, recognition or pattern algorithms have assumptions that can berelated to “probability” of the feature been searched. For example inobject recognition, if 4 out of 5 features for this object (face, lamp,chair) have been recognized, a probability P of 80% is indicated. Otherexamples may be comparison to a similar image for same scenerecognition, as objects have moved or just look similar. Aesthetic fitalgorithms employ “probability” which pertains to this.

Confidence Level of Quality Indicator

Thus far, only a current image has been illustrated, but previous datacan aid in computation of a better confidence level. This depends on theQI in question and the nature of the errors, but assuming a generalcase, one way of computing the confidence level of QI_(i) denotedC_(i)(t) is

C_(i)(t) = P_(i)(t_(j 2)) * N(P_(i)(t_(j 2)))${N(x)} = e^{\frac{- {({x - \mu})}^{2}}{2\sigma^{2}}}$${\mu_{i} = {\sum\limits_{k = {t\; 1}}^{t\; 2}\frac{{P( t_{k} )}_{i}}{{k\; 2} - {k\; 1}}}};{\sigma_{i} = \sqrt{\frac{1}{n}{\sum\limits_{k = 1}^{n}( {{P( t_{k} )}_{i} - \mu_{i}} )^{2}}}}$t_(j2) is the time of frame j2, the last frame so far.t_(j1) is the time of minimum frame j1 computation which may be used.

N(x) is a formula for a normal distribution, σ the standard deviationand μ the mean value. The values in the distribution here are P_(i)(t)

In the formula for computing normal distribution and standard deviation,there is a limit on the “looking back” portion of the computation, sowhen a significant event occurs at frame j1, it may be reset.

This example uses normal distribution, since nothing is known about theQI at hand, but this need not be the only distribution function. Forexample, if the location is measured by an accelerometer, accelerometerreadout errors and drifting errors over time may be used to construct abetter distribution function. For example, an accelerometer distributionfunction may be different for high acceleration and for lowacceleration, since in low values the distribution will be more spreadout than in high values.

Mean μ_(i) and standard deviation σ_(i) are computed from a specificframe j1 until the current time (or just until the previous frame). Thisway a significant event in j2 can be an indication to reset j1 to j2, sothe computations will not use frames before j2.

In another embodiment, one that doesn't use old data N=1, thusC _(i)(t)=P _(i)(t _(j2))

There may be other methods to compute confidence. For example, GPSlocation algorithms (such as a Kalman filter) have their own way ofcomputing confidence in the location, especially but not limited to, themean of error in position. If current GPS data say location has changed3 meters in the last 15 milliseconds, but accelerometer data shows nochange, and assuming that it is known from past computations that speedis relative zero, then the GPS data may not be trusted, and confidencein the GPS QI for movement, is low.

In another example, it may be that the lens module 200 report image isin focus, however independent focus blur QI 244 image is not in focus.This can affect the confidence of QI in component 248, where lens moduleQI is computed.

One way to compute the confidence level that is dependent on othervalues can be expressed as

${C_{i}(t)} = {{P_{i}( t_{j\; 2} )}*{N( {P_{i}( t_{j\; 2} )} )}*{\prod\limits_{k = 1}^{n}{f_{ik}( {t,{{QI}(t)}_{k}} )}}}$

Where n is the number of available quality indicators, and i and k arerunning indices of the current quality indicator in question. Thefunction ƒ_(ik) denotes a function specific to these quality indicatorcorrelations. If there is no correlation the function will be regardedas the scalar 1. In the above example, i being QI in component 248, andone some value 1≤k≤n; k≠i where k is the index for QI in component 244,the value of f_(ik) can be 0.5 if QI(t)_(k)<0.7. (In this example all QIand confidence levels are normalized to 1).

More examples of use and computation of confidence levels are nowdescribed, as other components and quality indicators are introduced.

Quality Indicators Use of Previous Frames Data

The QI used in the total quality indicator, QI_ForTotal(t)_(i) employsthe current value of QI_(i)(t) but also the confidence level. Theconfidence level is a function of old values inherited by virtue ofdistribution formula N(t). So if the QI value is significantly differentthen expected, the QI value's confidence level may drop therebybypassing a current QI that may be problematic.

Other ways to bypass or wholly or partially discount a current waywardQI value include, say, recognition of an object or face. The object mayhave been in the field of view one frame ago, with high confidence, say1/30 of a second ago. Now the QI say the object is no longer there,perhaps with low P_(i)(t) value. It is possible however that the objectin fact is still there; just the recognition algorithm has misses theobject. In this example the method may override the current QI with theprevious one, at least for a frame or two. More generally, old QI may beemployed, particularly if its confidence level exceeds newer QI values,to generate an improved current QI value. One embodiment could be anaverage of the last few frames that had a high confidence level. Theavailability of old QI, errors in them via P_(i), and the computation ofC_(i) provide other methods for determining the value of QI_(i)(t) e.g.for use in the total indicator. Algorithms of deep learning in neuralnetworks, applied to confidence level data inter alia, may be used tobetter compute a current value of the quality indicator.

The following are examples of quality indicators and computationalfunctionalities therefor, some or all of which may be provided as shownin FIG. 1:

Blur Detection in Component 244

Image blur can be of 2 types: focus or movement. In order to detectblur, quantify it and alert the user, its origin may be determined, sothe user will have information on why the picture has inferior quality.

Blur caused by movement can be either due to the imaging device havingmoved (usually shake) and/or because the object which was photographed,moved. Both movements refer to the time the image is taken, and theshutter speed.

There are many algorithms for testing blur via frequency distribution ink space transformation such as DCT or other Fourier transform variants,for example prior art US20110096180. Other ways may be testingsignal-to-noise ratio and searching for peak signal-to-noise ratio(PSNR). These algorithms can be used to create a blur quality indicatorin component 244. This may assist in double checking the camera module200 own focus assessment used in computation of QI in component 248.This is an advantageous step, since many quality indicators functionbetter when they use previous frames, assuming they are all in focus.Retesting lens/camera module 200 own focus assessment may therefore behelpful.

Device Movement Indication Based on Accelerometer

Camera shake and movement can be quantified using an accelerometer 206to measure device 3D acceleration, and integrating this over time todetermine camera speed. The time of resetting the speed to zero isadvantageous, since computation rounding and sensor errors mayaccumulate over time. This can be done in a few ways, for example: zerospeed can be accessed when the device's acceleration is minimal and backand forth over a long time (meaning device is around some point inspace).

Device Movement Quality Indicator 270

A person of skill in the field of photography would recognize the ruleof thumb of focal length which may be reciprocal to the shutter speed ifone wants to avoid camera shake due to hand shake. Component 200provides these two parameters. Device/surrounding objects 4D positioncomputation component 230 provides a position of the device as afunction of time in some degree of confidence. Given an average handjitter speed, one can set a threshold of the accepted speed of thedevice. For example, assume that for a shutter speed 1/15 of a second,speed may be less than 2 mm per second. 1 mm per second may provideexcellent quality, 2 mm per second good quality and 2 cm per second verypoor quality. Also a depth of field of the picture (DOF) can to becomputed, using the focus distance, aperture of the lens and so on. Ifthe depth of field is small and subject distance is short (macrophotography), then movement of the device in the z axis may have aprofound effect on quality but for most pictures and devices this is notthe case.

For example, a scene with a baby taken from 50 cm away with a NikonD300, 50 mm focal length lens, aperture of 1.8, means the depth of fieldmay be 2.7 cm. This means that the baby's face may not be fully infocus, and an inferior quality may be indicated. For the sameconfiguration with f/8 aperture, an excellent quality may be indicated.These computations may aid in determining the importance of movement inthe Z exis to the value of the QI. Let a function I₃(x) be theimportance of the z axis. In one embodiment I₃(x) may be an exponentialfunction of some sort. The other axes, x and y, each have their ownfunction. They can also have an exponential function, but with adifferent factor. Given the speed V_(i) in each direction as given byDevice/surrounding objects 4D position computation component 230 thefollowing can be computed:

${QI}_{{device}\mspace{14mu}{motion}} = \frac{\sum\limits_{1}^{3}{{I_{i}( x_{i} )}*{V_{i}(t)}}}{3}$

This is one possible method for computation of the base QI for devicemotion which is not intended to be limiting. Other embodiments may forexample use confidence levels.

Component 200 may have some kind of image stabilization—sensor shift,lens stabilization or other. In this case the lens or sensor movementspeed in these image stabilizations may be incorporated into the speedV_(i)(t) in the formula above, for the QI to be correct. If this cannotbe achieved with sufficient accuracy, the errors may be incorporatedinto this QI confidence level.

Object Movement Component 220

There are many methods of computing a motion vector. A block matchingmethod may be used, or a gradient method, etc., or if the motion vectorinformation is included in the input image signal in some form, thisinformation may be utilized. The motion of the objects in the imageframe is caused by the combination of two factors: real object movementand apparent movement caused by device movement.

When computing object speed in the image frame, one must subtract thedevice movement as computed in Device/surrounding objects 4D positioncomputation component 230 from the object speed vector computed.

Methods of statistical pattern recognition may be used to fullyrecognize actual objects, such as a chair, a tree, a face or a car. Ifeither the imaging device or object is moving, the identity of theobject may be considered as a factor for the related confidence level;e.g. a car is much more likely to be moving during imaging than is atree. Objects that are in the focus rectangle or area of interest aremore significant to quality. If there is some movement at a far endregion (or: at a significant distance), this may not greatly affect thequality indicator.

Object Movement Quality Indicator 260

The data from component 220 can be used to create a quality indicatorabout blur due to subject movement. Normally, no movement would indicateexcellent quality, and high-speed movement would indicate low quality.

Some objects are more important than others for a “good” picture. Forexample, often a face is more important than a chair. In order tocompute this subject movement quality indicator, each object may berecognized and its movement vector and significance may be factored intothe quality indicator dependent on this object's movement.

This computed data may be used to alert of motion of a specific objectthat is considered important such as a face located in the center of animage. One way of computation of this quality indicator, notedQI_(subject movement), may be:

${QI}_{{subject}\mspace{14mu}{movement}} = \frac{\sum\limits_{n}{{A( x_{i} )}*{B( x_{i} )}*{S( x_{i} )}}}{n}$where A(x)—binary/multi-value indicator: is object inside or outside of;within, close to or far from, the focus/interest area rectangleB(x)—importance of object to a pictureS(x)—object speed factor. A desired speed may be assigned a “high” valueof 1, and 0 may be assigned for non-desired speed, with possible valuesin the middle. If the object is not to look blurred, it should bestanding still relative to the current shutter speed. In that case ahigh value may be a slow speed. However, speed of an object may bedesirable, e.g. to create the artistic impression of movement in thepicture. In such a case, a high value may be assigned to a high speed.Each factor is normalized to 1 in this example.

In one embodiment, if the motion vector's estimation quality is notsufficient, the camera module may be instructed to take a picture of adifferent exposure, say a shorter one, to measure the movement moreprecisely.

Computing Object Motion Vector via Corners Detection

Algorithms such as Harris corner detection or its variants such as ShiTomas corners detection may be used to determine whether the new imageframe has more or less details than the previous frame. Thisdetermination may be combine with data from the gyroscope andaccelerometer, if provided, to indicate whether or not the device hasmoved and/or the quality has severely deteriorated or increased.

The total quality indicator can be shown to the user in real time. Also,given a configurable “minimum” quality, an indication to the user can bemade if the image total quality indicator is below this minimum.

In each region, the number of corners are counted and their center ofgravity is computed. This is done for each region on each frame. Thepath of each center of gravity may then be followed over each region anda motion vector may thereby be computed. This may be feasible if thespeed of the object is slow, since when the speed is quick, the cornersmay become blurred.

Image Leveled to Horizon Quality Indicator 271

The device gyroscope 205 and accelerometer 206, both of which may or maynot be provided or available, may be used to compute device angle to thehorizon, or how the device is leveled. Using this data, a qualityindicator for “device leveled according to its orientation”, iscomputed. For example, zero to 3 degrees is “excellent” quality, 3 to 5degrees is “good” quality and so on. The user may hold the device eitherhorizontally or vertically.

Camera Focus Quality Indicator 248

As already mentioned, component 200 may send to component 248 data ifcomponent 200 is currently in focus, thereby to form an “excellent”quality indicator level for already-focused and “bad” quality indicatorfor still-focusing. Some camera modules may provide a focus estimationof their own. In such cases, this may be incorporated into or replacethe quality indicator to provide more than a yes/no quality indication.

Furthermore, when the lens focuses, the frames received from the cameramodule are blurred, so the quality indicators related to the blur orobject motion detection may be affected. In one implementation, theframes with low focus quality, may be removed when computing thesequality indicators.

The lens/camera module 200 may also report its own focus confidencelevel.

Ideally, the focus indication sent by component 200 may coincide withcomponent 244 QI. However, if it does not, say 244 indicates out offocus and 200 indicates in focus, this can be used to lower 248confidence level, as demonstrated in the confidence level section.

Face Detection Quality Indicator 249

This quality indicator indicates whether there is a face or faces andwhether the face is smiling and/or looking at the camera. Theseindicators are passed on to a component 249 that analyzes the relevanceof each of them, and yields a “total” quality value for this qualityindicator.

Furthermore, one may determine if each face is smiling and/or looking atthe camera. This data may be used for a quality indicator. For example,a scene in which over 90% of the faces are looking at the camera and 80%are smiling, equals “excellent” face quality, while a scene in whichnone are looking and none are smiling indicates “bad” quality.Importance of the smiling/not smiling factor may be configured by theuser via component 151.

Recognized Face Quality Indicator 250

The device may have a list of configured faces to recognize and foreach, pre-stored, their importance, say a user family member or workcolleague. The data may also include specific face features. This isdenoted as component 212,

Recognition of these faces in component 217 may be used by component 250to produce a quality indicator based on importance as well as facequality, as before. Face importance may change over time by the user, orautomatically, by learning user preferences. Confidence level may becomputed similarly to 249. However, since it is a known person, moredata may be obtained to enhance face recognition, such as face featuresfrom various angles, glasses, etc.

ISO Level and Camera Hardware Exposure Data 245

Data such as ISO level, exposure time, focal length, and aperture andfocus area is sent from the lens/camera module 200.

Depending on the device capabilities, such as maximum and minimum ISO, aquality indicator is produced. High ISO gives bad quality vs low ISOthat gives a high quality indicator.

An optional exposure time quality indicator may be created, if a minimumexposure time is configured.

Lens Focal Length, Aperture, Focus Distance, Sensor Type and Model andDepth of Field 247

Depending on the device capabilities to change focal length andaperture, and given a known configured scene, a quality indicator may beprovided to indicate how these parameters confront the given scene. Forexample, if the scene is a portrait and no face is detected by component216 yields a bad QI.

Another example may be to use the sensor type and model (component 200)to yield a QI related to it—professional quality may yield good quality,when a poor quality, less costly component, yields poor quality.

Another embodiment for this QI is for testing background blurring, whichmay sometimes actually be desired. For example, a scene with facerecognition in 216 and a very small aperture, say f10 with focusdistance in the hyper focal distance of component, 200 yields a badquality, whereas a wide aperture, say f2.8 on these conditions, yields agood quality.

Under and Overexposed Quality Indicator 243

Given configurable under and over thresholds (say 5% from pure white and3% from pure black) the image frame given to the GPU is processed tocompute areas that are underexposed and areas that are overexposed, e.g.similar to the zone system by Ansel Adams known in the field ofphotography.

The image frame is divided to a configurable number of areas; say 256areas or 1024 areas. If a configurable number of percentages of pixelsin an area (say 95%) are under or over exposed, the area is marked asunder or over exposed. If the image sensor or operating system supportsscene recognition/scene mode, or object recognition is present, then theimage sensor or operating system may be used in the evaluation of thisquality indicator. For example, if the sun or a lamp is the brightestarea in the image, they may be ignored. A point of bright light like thesun or a light bulb may be recognized by their pattern and may beignored.

On another embodiment, the computation may be based on percentage ofover and underexposed pixels in the image.

An image in which a large percentage is marked as over or under exposed,say over 40%, may get a bad over/under quality indicator. The samethresholding method may be used for the method of areas described beforee.g. if over 40% of 256 areas are considered over exposed.

In another embodiment, pixel or areas inside or outside the focus areaof the picture, as given by the lens/camera module 200, may have adifferent weight e.g. pixels or areas outside the focus area may have alower affect on the quality indicator value.

This indicator may be used to indicate to the device that an Highdynamic range (HDR) shot may be taken, and may be done eitherautomatically or by notifying the user. Depending on the device, thismay be done automatically by taking 2 pictures with different exposuresettings.

Histogram Quality Indicator 241

The under/over quality indicator is a special case of a more generalquality indicator taking into account not just the histogram extreme,but also its central part, and also taking into account functionstructure and lens/camera module 200's available dynamic range.

This may be no reference, or a full reference (meaning in computationthere is or there is not a reference to a preferred image), e.g. as in“The Design of High-Level Features for Photo Quality Assessment” by YanKe, Xiaoou Tang and Feng Jing.

Obstructing the Lens Quality Indicator 246

Obstructing the lens may occur as a result of putting the finger partlyor fully on the lens. For example, in bright conditions there may be abrown-red area in one corner. In dark conditions there may be no lightat all in that location. Confidence in having recognized the situationmay be enhanced as more frames are taken and the camera is moved aroundbecause if there is no obstruction on the lens, then the size of thearea previously believed to be obstructed, would change significantly.There may be other objects causing obstruction that may be recognized.If they are recognized, their name can be used in feedback to the user,instructing him to remove the obstruction.

The amount of area and confidence may be used to give this qualityindicator a degree/rating between “excellent” and “bad”. For example, afinger blocking the lens may look on the image as a brownish area,probably in a corner of the frame. If the color were green, or it is anisolated area in the middle, the confidence would be low.

For the previous example (finger blocking appearing as a brownish area)the confidence is high.

Functionality 230—Measuring Distance to Objects and Creating a 4DEnvironment Measurement in Real Time e.g. by Computation ofDevice/Surrounding Objects 4D Position

As already described, the speed of the device in each direction may bemeasured and the distance traveled may be computed.

With the help of a GPS (e.g. based on unit 202 in FIG. 1) and/or stepcounting hardware (unit 203) accuracy in the range of a few dozen or adozen or centimetres or even a single centimetre may be computed by aperson skilled in the area for the device 3D relative position in time,thereby to yield 4D position.This could be used by component 270 to yield better QI accuracy. Bymeasuring the boundaries of an object in one location and again inanother location, after moving the device, the distance to the objectmay be measured (assuming the object is still). e.g. via parallax.Parallax is the apparent displacement of an object because of a changein the observer's point of view. Since cameras today easily achieve 8 MPand more, typically, distance can be computed providing parameters suchas lens focal length from component 200.

The 3D scene is built according to the input images from the imagecapturing unit 200 or any other sources. It can be based on multi-viewimage geometry techniques as introduced in “Multiple View Geometry inComputer Vision” by R. I. Hartley, A. Zisserman, Cambridge UniversityPress (2004). The user may help by moving the device, either by walkinga few steps or holding the device at full arm's length and swinging itpointing to the scene in question. However, a full 3D scenereconstruction is not required. A partial reconstruction, includingscene change in time, may suffice.

Since the methods used to estimate the distance include integratedestimation, such as the distance is between 2 to 3 meters, a confidencelevel on each object can be constructed, similar to that described inthe confidence level section. Also, since the device position itself mayhave errors or uncertainties, this also can be used in the confidencelevel of each object. This confidence level can then be used by qualityindicators using this component data, so that they may implement theirown confidence level.

Aesthetic Rules Quality Indicators 240

In photography, there exist ‘golden rules’ for fitting of thecomposition to a section that may give a better aesthetic lookingpicture. There are heuristics that provide a means of achieving aneye-pleasing composition such as but not limited to: rule of thirds,golden triangle, golden spiral, shapes and lines, amputation avoidance,visual balance, and diagonal dominance.

For example, when taking a picture on a beach, the sea and sky meet onthe horizon ‘line’. To fit this rule, one must put the line either atone third-height or two-thirds height, thereby fitting the “rule ofthirds”. Either one is “correct”. However, putting the line in themiddle creates a composition that is considered less favorable to theviewer.

There are various methods of testing a fit to this rule, for example in“Photographic”, Volume 29 (2010), Number 2 “Optimizing PhotoComposition” by L. G. Liu, R. J. Chen, L. Wolf, and D. Cohen-Or.

Further types of aesthetic models exist, based on chroma and exposure.Some may only give a non-reference quality indication without suggestionon coordinate change, as in the previous article. For example, a modelis described in published US20130188866.

In another embodiment, multiple aesthetic qualities may be used.

This aesthetic QI, can be used for suggestion to the user to move fromhis current location to another location, together with the otherquality indicators like undercover, it may be suggested to the user thathe should move from his current location.

For example, a lamp in the street behind a person at night will not begood even if it fits a golden rule. Using data from Device/surroundingobjects 4D position computation component 230, we can assess distance tothe person's face, say 2 meters, the distance to a lamp of 4 metersheight 4 meters behind him, we can suggest to the user that he shouldmove 2 steps to the right and one forward, tilting the device 20 degreesto the left if he wishes to achieve a better aesthetic QI score and abetter total QI.

Functionality 242: Same Scene Quality Indicator

One of the problems solved by certain embodiments of the presentinvention include a processing system to reduce or eliminate any need totake multiple pictures of the same scene, in order to get one goodpicture. If the proposed system negates this need, it may be helpful toindicate to the user he is taking unnecessary pictures that may only bea burden at a later stage.

This is a heuristic quality indicator, taking into account the time,location of device, angle of device, object movement in the device andother parameters, including but not limited to photographic parameterssuch as ISO and focal length. It may require these parameters ofprevious pictures taken in recent time. One possible implementationincludes comparing each QI and its confidence data to the QI andassociated confidence level of a previously recorded and saved image inthe last say 5 seconds (configurable). If the data of each QI is thesame, it may be concluded that it is a “bad” same scene qualityindicator.

Functionality 400 and 401: Combining the Separate Quality Indicators toTotal Quality Indicators

As described herein, some quality indicators and their confidence levelsmay depend on other QI data. For example, aesthetic QI computation 240may need the data on image leveled to horizon from component 271. Ifconfidence of 271 is very low, it may not use any other data. Some, butnot limited to, examples of QI computation components can be seen inFIG. 3.

Some quality indicators may be considered mandatory (unless specificallyturned off) since they cause unacceptable blurring of the image:

-   -   Focus    -   Subject movement    -   Device movement        Each device may have different hardware capabilities, thereby        may not yield all the possible quality indicators or may not be        able to process detail indicators, due to slow hardware. Some        indicators may be ignored, if so configured by the user. One is        then left with a set of indicators with which to compute.

A QI use in a total indicator may vary depending on situation. Forexample, “overexposed” quality indicator computation 243 may use datafrom object recognition in component 220. If the picture shows a sunset,it is likely that there will be over and underexposed areas, so in oneembodiment it may be chosen to decrease importance of a sun-lit area andbe less sensitive to overexposed QI in total computation. In anotherexample, face exposure check may be increased if the face qualityindicator suggests there is a face.

${{QI\_ ForTotal}(t)_{i}} = {{{QI}(t)}_{i}*{\prod\limits_{j = 1}^{n}{f_{ij}( {t,{{QI}(t)}_{j},C_{j}} )}}}$As can be seen in this formula, there is a dependency in the confidencelevel C_(j) before it is used, so data of QI with low confidence levelmay not be used. There may be other ways to formulate these connections.This may be employed as a general case.

${QI}_{total} = {\sum\limits_{i = 1}^{n}{{QI\_ ForTotal}(t)_{i}*{w( {t,C_{i}} )}_{i}}}$W(t)_(i) is the weight of each QI in the total and may be used to ignoreQI if desired. W(t)_(i) is dependent on time, since the user may chooseto change it. W(t)_(i) may be dependent on the confidence level. Forexample, if it is not desired to include below say 10% confidence, andw_(i) will be zero if C_(i)<0.1.There can be more than one type of total QI. For example, when computingtotal QI, and only mandatory QI are chosen (QI's considered mandatory byan application or use case), a mandatory total QI may be provided. ThisQI may be used to decide when to take the picture, e.g. as describedbelow. Another option is to use all the QI to get a sense of the picturequality as a whole. One can choose to formulate other total QI for otherneeds.This formula is just one way to achieve a total indicator that isdependent on quality indicators, their confidence levels and theirrelations. The idea is to enable a general formula as quality indicatorsare added or removed in some embodiments, or for differentimplementations with various levels of complexity and interconnectionsbetween QI.

Functionality 502: Taking a Picture at the Proper Time Using TotalQuality Indicator

It may be that the picture will not be taken when the shutter ispressed, but only when a minimum threshold on specific QI with specificconfidence level is met, when a total QI minimum threshold is met, ortimeout occurs.

In such cases, the user may be shown, in real time, the QI values andpossible suggestions, as indicated in 501. The data for the suggestionsmay be in 520, where QI and their confidence level may be used to showsuggestions. Suggestions can be shown for QI how a value is no good, andconfidence level beyond a threshold, say 0.7.

The QI indication user interface may be a number, and may even be anicon representing the quality. For example 5 smiles may each represent aquality indicator. Smiling face—good, crying face—bad, and so on.

A picture, once taken, may be shown to the user with its qualityindicators, together with further suggestions on how to improve (forexample “image is underexposed in a large area”).

The image may not be saved or may be deleted if total QI or any otherpredetermined QI is below some quality indicator threshold. After theimage is saved, the quality indicators may be saved along with the image(one embodiment may be to use the EXIF convention used in JPEG files) sothat they may be used at a later stage, such as when a user wants to seethe picture quality days after it was taken.

Suggestions to the User on How to Improve the Picture and Component 520

The user may choose to get suggestions from the application on how toimprove the next shot. The application may use the quality indicatorsand their correlation to answer that.

This can be done before a final picture is taken and/or after. Theapplication may use quality indicators, their confidence level,importance and their correlation to respond. This is where the databasewith user suggestions, component 520, is used. The data in this can be atable of text to be shown for every given QI with a low value. In a morecomplex embodiment, it can be an SQL table being queried by multiple QI.An example of such a query on the database may be “select * fromsuggestion_table where QI1>0.9 and QI2<0.2” In this example QI1 can bemodule 200 ISO speed and QI2 can be device shake QI. The text retrievedcan be “Try to use higher ISO speed to counteract device shake”. Anotherexample can be QI1 is QI of 248 and QI2 is of 244. The text retrievedcan be “focus may be off. Focus one another subject and try again”.There may be more complicated queries and embodiments, as a person inthe field of databases can easily construct, given ideas from a personin the field of photography.

Text suggestions may be accompanied by an image explaining them. Forexample, if there is a low quality indicator of under/over exposure, theimplication may be presented to the user via text, and overexposed areason a picture preview may be painted in red, and underexposed areas maybe in blue.

The confidence levels may aid in recognizing the best suggestion to beretrieved from a table of suggestions. For example, one my use a QI withconfidence larger than 0.5, and show suggestions in the order ofconfidence levels. In another example, certain QIs may always havepriority over others, even if their confidence levels are lower thanothers, as long as they are higher than, say 0.7.

Device Movement Gestures as Indications for Shutter Press and Delete

In another embodiment the user may not need to indicate he is ready byfirst pressing the shutter. In such cases, the device may assume thatthe user is ready at any time, or may wait for the device to bestationary hence pointed at some specific point in space for a certainconfigured time. For example, the user may pull the device out of hispocket and point it to the desired scene and hold it there for aduration of one second. The device hardware may detect that the devicehas been in motion for a few seconds, and then held relatively still inspace, pointing at a certain direction for say one second. The systemmay assume that the user pulls the device out of his pocket in order totake a picture, and thus goes into “take picture” mode, waiting for thequality indicators to be above the threshold, then taking the pictureautomatically. The user may hear the sound of a picture being taken andreturn the device to his pocket. A shake gesture (available in mobile OSlike iOS) may be an indication to the device to delete the picture andstart again. Alternatively, the shake gesture may be recognized via thedevice shake QI.

Using the System to Evaluate Video Quality

A quality ranking system over continuous frames, may be used to evaluatea video comprising said frames. One method would be to compute andaverage the total quality indicators over frame.

In another embodiment a different set of weight functions w_(i) may beused e.g. different weights for different frames.

Further Examples of System Use are as Follows:

-   -   a) If device shake QI is bad or camera focus QI is bad,        disregard use of aesthetic QI in total computation, even if user        has indicated he would like to give it a high priority. This        will prevent the device from taking a picture when the picture        is blurry. The suggestion to the user can be “Hold still” or        “wait for focus”. Once shake is low and focus is optimal,        aesthetic QI in total quality computations may be used.    -   b) Quality reported by lens module 200 showing focus is fine,        but independent focus blur QI 244 shows the lens is completely        out of focus e.g. due to a poor lens module focus application.        Therefore, confidence in the lens focus quality indicator drops        and the weight of this indicator in the total quality may be        reduced.    -   c) Device movement QI confidence level computed in 270 is low        due to errors in computing location in Device/surrounding        objects 4D position computation component 230, or to sub        standard accelerometer, but face recognition system 216 with        excellent confidence level, insists on a face for 10 frames        straight. In this case, use face recognition QI in total QI even        if shake in 270 is high, but only at a reduced percentage of,        say, 50% of normal face QI.    -   d) Device movement importance is high if the depth of field of        the lens is very small (for instance, in macro photography). For        a quality picture, high confidence and low value of device shake        QI in 270 are desirable, and their weight in the total QI may be        high. User suggestions can be to hold still or use a flash, for        example.    -   e) If previous frames indicated a face in the far left or right        of the picture, and now there is no face, it is very possible        that the picture has “half a face”, amputation, so that may be        the feedback to the user. For example, a face has been        identified by the face recognizing QI, but attributes low        confidence as a result. The system can alert the user of a        possible face amputation.    -   f) A person may want to take a picture of, say, his spouse in a        scenic location, or near a historic building. He may choose to        give aesthetic QI and recognized person a high priority. This        priority may enable to build a total QI with that weight, and to        instruct the photographer to move a few meters to the right and        left for example, to get an approximated 3D environment in        Device/surrounding objects 4D position computation component        230. The device may measure all QI, especially the aesthetic QI,        and may instruct the photographer on the best location to stand.        This may depend on the place with the best total QI.

The system may if desired be implemented as a web-based system employingsoftware, computers, routers and telecommunications equipment asappropriate.

Any suitable deployment may be employed to provide functionalities e.g.software functionalities shown and described herein. For example, aserver may store certain applications, for download to clients, whichare executed at the client side, the server side serving only as astorehouse. Some or all functionalities e.g. software functionalitiesshown and described herein may be deployed in a cloud environment.Clients e.g. mobile communication devices such as smartphones may beoperatively associated with but external to the cloud.

It is appreciated that terminology such as “mandatory”, “required”,“need” and “must” refer to implementation choices made within thecontext of a particular implementation or application describedherewithin for clarity and are not intended to be limiting since in analternative implantation, the same elements might be defined as notmandatory and not required or might even be eliminated altogether.

It is appreciated that software components of the present inventionincluding programs and data may, if desired, be implemented in ROM (readonly memory) form including CD-ROMs, EPROMs and EEPROMs, or may bestored in any other suitable typically non-transitory computer-readablemedium such as but not limited to disks of various kinds, cards ofvarious kinds and RAMs. Components described herein as software may,alternatively, be implemented wholly or partly in hardware and/orfirmware, if desired, using conventional techniques. Conversely,components described herein as hardware may, alternatively, beimplemented wholly or partly in software, if desired, using conventionaltechniques.

Included in the scope of the present invention, inter alia, areelectromagnetic signals carrying computer-readable instructions forperforming any or all of the steps or operations of any of the methodsshown and described herein, in any suitable order including simultaneousperformance of suitable groups of steps as appropriate; machine-readableinstructions for performing any or all of the steps of any of themethods shown and described herein, in any suitable order; programstorage devices readable by machine, tangibly embodying a program ofinstructions executable by the machine to perform any or all of thesteps of any of the methods shown and described herein, in any suitableorder; a computer program product comprising a computer useable mediumhaving computer readable program code, such as executable code, havingembodied therein, and/or including computer readable program code forperforming, any or all of the steps of any of the methods shown anddescribed herein, in any suitable order; any technical effects broughtabout by any or all of the steps of any of the methods shown anddescribed herein, when performed in any suitable order; any suitableapparatus or device or combination of such, programmed to perform, aloneor in combination, any or all of the steps of any of the methods shownand described herein, in any suitable order; electronic devices eachincluding a processor and a cooperating input device and/or outputdevice and operative to perform in software any steps shown anddescribed herein; information storage devices or physical records, suchas disks or hard drives, causing a computer or other device to beconfigured so as to carry out any or all of the steps of any of themethods shown and described herein, in any suitable order; a programpre-stored e.g. in memory or on an information network such as theInternet, before or after being downloaded, which embodies any or all ofthe steps of any of the methods shown and described herein, in anysuitable order, and the method of uploading or downloading such, and asystem including server/s and/or client/s for using such; a processorconfigured to perform any combination of the described steps or toexecute any combination of the described modules; and hardware whichperforms any or all of the steps of any of the methods shown anddescribed herein, in any suitable order, either alone or in conjunctionwith software. Any computer-readable or machine-readable media describedherein is intended to include non-transitory computer- ormachine-readable media.

Any computations or other forms of analysis described herein may beperformed by a suitable computerized method. Any step described hereinmay be computer-implemented. The invention shown and described hereinmay include (a) using a computerized method to identify a solution toany of the problems or for any of the objectives described herein, thesolution optionally include at least one of a decision, an action, aproduct, a service or any other information described herein thatimpacts, in a positive manner, a problem or objectives described herein;and (b) outputting the solution.

The scope of the present invention is not limited to structures andfunctions specifically described herein and is also intended to includedevices which have the capacity to yield a structure, or perform afunction, described herein, such that even though users of the devicemay not use the capacity, they are if they so desire able to modify thedevice to obtain the structure or function.

Features of the present invention which are described in the context ofseparate embodiments may also be provided in combination in a singleembodiment.

For example, a system embodiment is intended to include a correspondingprocess embodiment. Also, each system embodiment is intended to includea server-centered “view” or client centered “view”, or “view” from anyother node of the system, of the entire functionality of the system,computer-readable medium, apparatus, including only thosefunctionalities performed at that server or client or node.

Conversely, features of the invention, including method steps, which aredescribed for brevity in the context of a single embodiment or in acertain order may be provided separately or in any suitablesubcombination or in a different order. “e.g.” is used herein in thesense of a specific example which is not intended to be limiting.Devices, apparatus or systems shown coupled in any of the drawings mayin fact be integrated into a single platform in certain embodiments ormay be coupled via any appropriate wired or wireless coupling such asbut not limited to optical fiber, Ethernet, Wireless LAN, HomePNA, powerline communication, cell phone, PDA, Blackberry GPRS, Satelliteincluding GPS, or other mobile delivery. It is appreciated that in thedescription and drawings shown and described herein, functionalitiesdescribed or illustrated as systems and sub-units thereof can also beprovided as methods and steps therewithin, and functionalities describedor illustrated as methods and steps therewithin can also be provided assystems and sub-units thereof. The scale used to illustrate variouselements in the drawings is merely exemplary and/or appropriate forclarity of presentation and is not intended to be limiting.

The invention claimed is:
 1. A method for estimating quality of at leastone image from a plurality of images, for use with a device thatcomprises in a single enclosure a digital camera module or functionalitythat comprises at least one optical lens for focusing received lightfrom a scene and an image sensor coupled to the optical lens forcapturing an image of the scene; a motion or location sensor for sensingthe device motion; and a processor coupled to at least one image sensorand to the digital camera for receiving data therefrom, the method bythe processor comprising use of at least one value and weight; obtaininga first value (QI1) responsive to the device motion from the motion orlocation sensor; estimating a first weight (c1) associated with thefirst; obtaining a second value (QI2) associated with the digitalcamera; estimating a second weight (c2) associated with the secondvalue; analyzing the captured image for detecting or recognizing one ormore objects in the image and obtaining a third value (QI3) associatedwith motion of at least one of said objects; estimating a third weight(c3) associated with the third value; obtaining a fourth value (QI4)associated with recognition of object characteristics; estimating afourth weight (c4) associated with the fourth value; obtaining a fifthvalue (QI5) associated with aesthetic quality of image based oncomposition; estimating a fifth weight (c5) associated with the fifthvalue; and calculating a total quality value according to, or based onvalues QI1, QI2, QI3 and QI5 and weights c1, c2, c3, c4 and c5.
 2. Themethod according to claim 1, wherein the estimating of the first weight(c1) is responsive to at least one of second, third, fourth or fifthvalue (QI2, QI3, QI4, QI5); or wherein the estimating of the secondweight (c2) is responsive to at least one of the first, third, fourth orfifth value (QI1, QI3, QI4, QI5); or wherein the estimating of the thirdweight (c3) is responsive to at least one of first, second, fourth orfifth value (QI1, QI2, QI4, QI5); or wherein the estimating of thefourth weight (c4) is responsive to at least one of the first, second,third or fifth value (QI1, QI2, QI3, QI5); or wherein the estimating ofthe fifth weight (c5) is responsive to at least one of the first,second, third or fourth value (QI1, QI2, QI3, QI4).
 3. The methodaccording to claim 1, wherein the first weight (c1) is estimatedaccording to, or based on, a precision error, reading resolution, ordrift in time, of the motion or location sensor, and wherein the motionor location sensor consists of, or comprises, an accelerometer, agyroscope, a Global Positioning System (GPS), or a step counter.
 4. Themethod according to claim 1, wherein the third weight (c3) is estimatedaccording to, or based on, the estimated error in the analyzing thecaptured image for detecting or recognizing objects in the image.
 5. Themethod according to claim 1, wherein the digital camera furthercomprises an auto-focus functionality, wherein the auto-focusfunctionality is coupled to the controlled by the processor, and whereinthe method further comprising controlling the auto-focus functionalityaccording to, or based on, the second value (QI2) or the total value. 6.The method according to claim 1, wherein object characteristics are atleast one of recognized smile, looking at camera, face importance touser or confidence of object recognition.
 7. The method according toclaim 1, wherein aesthetic quality of image based on compositionincludes at least one of: rule of thirds, golden triangle, goldenspiral, shapes and lines, amputation avoidance, visual balance, diagonaldominance, non-reference quality indication or relative position ofsalient regions in the image.
 8. The method according to claim 1,further comprising grading the image quality according to, or based on,the total value, and wherein the total value is calculated as a weightedaverage value according to, or based on,c1*QI1+c2*QI2+c3*QI3+c4*QI4+c5*QI5.
 9. The method according to claim 1,for use with a probability distribution function using a mean value (p)and a standard deviation value (a), wherein the estimating of at leastone of the weights (c1, c2, c3, c4, c5) is according to, or based on,the respective probability of the at least of the values (QI1, QI2, QI3,QI4, QI5), according to the probability distribution function.
 10. Themethod according to claim 1, for use with a plurality of former valuesof the respective QI, further comprising calculating the mean value (μ)and the as the standard deviation value (σ) based on the plurality offormer values.
 11. The method according to claim 1, wherein the actingcomprises affecting the digital camera in response to the total value.12. The method according to claim 1, wherein: the device comprisesmultiple motion or location sensors for sensing the device motionselected from a group consisting of a gyroscope, accelerometer, GlobalPositioning System (GPS), 9 Degrees of Freedom (DOF) sensing component,and 10 Degrees of Freedom (DOF) sensing component, wherein the methodcomprising obtaining a corresponding first value (QI1i) from each of themultiple motion or location sensors responsive to the device motion fromthe respective motion or location sensor; and estimating or calculatingthe first value (QI1) based on the multiple first values (QI1i) from themultiple motion or location sensors, or wherein: the analyzing of thecaptured image comprises applying multiple algorithms selected from agroup consisting of an aesthetic algorithm, an artificial neural networkemploying deep learning algorithm, a corner detection algorithm, a blurdetecting algorithm, and Peak Signal-to-Noise Ratio (PSNR) calculation,and the method further comprising obtaining a respective third value(QI3i) associated with each of multiple algorithms, and calculating orestimating the third value (QI3) based on the multiple third values(QI3i) from the multiple algorithms.
 13. The method according to claim1, wherein estimating fourth value (QI4) or forth weight (c4) is basedat least partly on a list of configured faces.
 14. The method accordingto claim 1, wherein algorithms of deep learning are used to compute atleast one of the values (QI1, QI2, QI3, QI4, QI5).
 15. The methodaccording to claim 1, wherein algorithms of deep learning are used tocompute the total quality value.
 16. The method according to claim 1,wherein the second value is a combination of at least one of digitalcamera exposure, focus and under or over exposure.
 17. A method forestimating quality of at least one image from a stream of images from avideo, for use with a device that comprises in a single enclosure adigital camera module or functionality that comprises at least oneoptical lens for focusing received light from a scene and an imagesensor coupled to at least one optical lens for capturing an image ofthe scene; a motion or location sensor for sensing the device motion;and a processor coupled to the image sensor and to the digital camerafor receiving data therefrom, the method by the processor comprising useof at least one value and weight; obtaining a first value (QI1)responsive to the device motion from the motion or location sensor;estimating a first weight (c1) associated with the first; obtaining asecond value (QI2) value is a combination of at least one of digitalcamera exposure, focus and under or over exposure; estimating a secondweight (c2) associated with the second value; analyzing the capturedimage via deep learning algorithms for detecting or recognizing one ormore objects in, or one or more characteristics of image or at least oneof object characteristics to obtain a third value (QI3) associated withthe analysis; calculating a total quality value according to, or basedon values QI1, QI2, QI3 and weights c1, c2, c3 and previous values QI1,QI2, QI3 and previous weights c1, c2, c3 in said image stream; andselecting by said processor at least one image from the plurality ofimages at least partly on said total quality value.
 18. The methodaccording to claim 17, where a first value (QI1) can be a combination ofaccelerometer data, gyro data and camera/lens module.
 19. The methodaccording to claim 17, wherein algorithms of deep learning are used tocompute at least one of the values (QI1, QI2, QI3, QI4, QI5).
 20. Themethod according to claim 17, wherein algorithms of deep learning areused to compute the total quality value.